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Problem Statement 

The gathering of feedbaek is an important part of edueational program development and workforee development in 
general. It is eritieal that human resouree development praetitioners are able to gauge the sueeess of their efforts 
from the viewpoints of a wide variety of stakeholders. One of the best ways to gather employee and eonsumer 
feedbaek for researeh, evaluation, and eontinuous improvement is through the use of surveys. Telephone surveys or 
hard-eopy surveys, mailed or manually distributed to potential survey respondents, have traditionally been ways 
researehers have gathered survey data. In the age of the Internet, however, online surveys ean be an effieient way to 
gather data as well (Straehota, Sehmidt, & Coneeieao, 2005). Despite the myriad of opportunities offered by the 
Internet, survey researehers have not been leaders in the use of Internet-based surveys. In faet, Dillman and Bowker 
(2001) note that “. . .the use of web surveys seems to have eaught the survey methodology eommunity somewhat by 
surprise. Leadership for the development of web survey proeedures has eome in large part from eomputer 
programmers, many of whom have little or no training in survey methodology” (p. 1). They further posit that “rather 
than being at the forefront of this latest innovation in the eonduet of soeial surveys, survey methodologists are 
playing eateh-up as they learn to master these new survey development tools” (p. 1). The purpose of this paper is to 
deseribe an effieient proeess for eondueting online survey researeh. The proeess is deseribed using two empirieal 
ease studies. 

Using the Internet as a data-gathering tool adds extra layers of eomplexity to the proeess of survey 
development. Gunn (2002) suggests that “unlike other types of surveys, web page design skills and eomputer 
programming expertise play a signifieant role in the design of web-based surveys” (p. 1). These ehallenges have 
been made somewhat easier by the development of websites speeifieally geared to help researehers develop and 
distribute online surveys. Websites sueh as SuveyShare.eom, SurveyMonkey.eom, and Zoomerang.eom allow 
researehers to set up online surveys in a matter of minutes. Many of these sites offer distribution, tabulation, and 
reporting features as well. However, researehers must be aware that there are additional eonsiderations assoeiated 
with the use of online surveys that must be addressed as part of survey development and distribution proeesses 
(Straehota et al., 2005). This paper examines two empirieal studies that used online surveys to eolleet data to 
measure satisfaetion in job training and workforee development. A deseription of eaeh study and findings related to 
response rate, the proeesses used in online survey development and implementation, as well as reeommendations for 
the future use of online surveys are presented. 

Theoretical Framework 

The theoretieal framework of this paper is based on the work of Wallen and Fraenkel (2001), Rea and Parker (1997), 
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Dillman, Tortora, and Bowker (1998), Dillman and Bowker (2001), and their contributions to survey development. 
Wallen and Fraenkel’s process includes the definition of the problem, identification of the target population, 
determination of mode of data collection, preparation of the instrument, collection of data, and analysis. Also 
important to their process is the identification of large categories of issues (or constructs), which can be used to 
suggest more specific issues within each category, which can then be used for question development. Rae and 
Parker’s (1997) stages of the survey research process were also used in theoretical framework development. 
Works by Dillman et al. (1998) and Dillman and Bowker (2001) form the theoretical bases for the online 
components of this work. These researchers have worked extensively in online survey development and 
distribution. 

Research Questions 

This study addresses the following research questions: 

• How can online surveys most efficiently be used to gather data for evaluation and research within the field 
of human resource development? 

• What are the pros and cons associated with the use of online surveys related to human resource 
development evaluation and research? 

Efficiency is an outcome of effective learning organizations. According to the American Heritage Dictionary (2001) 
efficiency is defined as “producing an effect with a minimum of waste and effort; exhibiting a high ratio of output to 
input” (p. 274). This paper will illustrate how online surveys offer the researcher time and cost efficiency, as well as 
methodological accuracy in the data collection process. According to Pande, Neuman, and Cavanagh (2000), “more 
efficient processes use less money, time and materials” (p. 203). Further, researchers must also be aware of the pros 
and cons of online survey development and distribution as they relate to the field of human resource development. 

Research Design 

This paper uses case studies to describe a process (Merriam, 1998). Case studies were used in part because of the 
dearth of information on online survey development and distribution. A review of literature shows evidence of 
online surveys used for research studies, but no detailed descriptions of how online surveys can be used most 
efficiently. Our goal in writing this paper is to contribute to the body of knowledge on online survey development, 
which will help practitioners and researchers in the field of human resource development. 

Online Snrvey Development and Implementation 

The development of an online survey instrument begins with the design of the survey itself While this process may 
not have changed with the advent of the web-based survey, factors in addition to those associated with traditional 
survey development should be considered when a survey is put online. Schmidt (2004) and Strachota (2003) 
followed the recommendations of Dillman, Tortora, and Bowker (1998) when developing their online surveys. 
Dillman et al. (1998) recommend that “questions must be presented in a way that they can be understood and 
answered accurately by all recipients. And, the likelihood of responding should not be affected by peoples’ lack of 
computer skills” (p. 3). To address these issues, Dillman et al. (1998) advise online survey developers to consider 
the following: 

• Survey design should be respondent-friendly, and take into account the inability of some respondents to 
receive and respond to web questionnaires with advanced programming features. 

• Survey design should consider both the logic of how computers operate and the logic of how people expect 
questionnaires to operate. 

• As many survey populations do not have access to the Internet, a survey design that allows for survey use 
in other modes of data collection (i.e., paper questionnaires, interviews, or telephone surveys) is important. 
It is also important to remember, from a reliability standpoint, that those other modes of data collection be 
administered using as many of the same guidelines as the online survey as possible (i.e., the timeframes in 
which the surveys are administered and the location at which surveys will be completed should be similar 
for both online and non-online respondents). 

Researchers must also consider issues associated with the distribution of web-based surveys to potential 
respondents. According to Timmerman (2002), “with web-based surveys, participants are usually notified by e-mail 
to participate in the survey. The e-mail generally includes a link to the URL (Uniform Resource Locator) web 
address of the survey” (p. 3). 
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Two Empirical Cases Employing Online Snrvey Research 

Both case studies described below featured the use of online surveys to collect data to measure either satisfaction in 
job training or in workforce development. Below is a description of the survey development and distribution 
processes used in each study. 

Job Training 

In the job training and job satisfaction study, measurements for job training satisfaction and overall job satisfaction 
were separated so each topic could be examined individually. Because it is a well-studied topic, it was decided to 
use an existing survey instrument to examine job satisfaction. The instrument chosen was one designed by Paul 
Spector (1997) which is the 36-item, nine-factor Job Satisfaction Survey (JSS). The constructs Spector (1997) used 
to examine overall job satisfaction included satisfaction with pay, promotion, supervision, fringe benefits, contingent 
rewards, operating procedures, coworkers, nature of work, and communication. There are four questions for each 
subscale. The job training facets of this survey measure employee satisfaction with on-the-job training. The three 
subscales (or constructs) measure organizational support for training, employee feelings about training, and 
employee satisfaction with training. Specific questions for each of these constructs were developed by Schmidt 
(2004) based on learning organization assessments found in “The Learning Edge” (Wick & Leon, 1993). Four 
questions were developed for each subscale. Additionally, the Job Training and Job Satisfaction Survey contains 
five questions regarding training methodology, training content, and time spent in training. In addition to separate 
subscale use, the entire measure may be used as an overall measure of training satisfaction. The job training 
constructs, which included the definitions and questions for each construct, were examined for validity at this point 
in the instrument development process by subject-matter experts in the field of adult education, and by a panel of 
measurement experts. 

In order to ensure construct validity, a pilot study (n=118) was conducted, and the internal consistency of the 
two scales (job training and job satisfaction) was assessed. Cronbach’s alpha was found to be .83 for the job 
training scale and .89 for the job satisfaction scale. Based on data collected in the pilot, inappropriate questions 
were deleted, three of Spector’ s original subscales were combined into one, and one construct from Spector’ s 
original survey was deleted. The remainder of Spector’ s original subscales engendered a relatively clean factor 
structure with limited overlap between the scales. 

In the actual study, a convenience sample of 552 customer and technical service employees in nine major 
organizations in the United States and Canada were given the Job Training and Job Satisfaction Survey. 
Organizations chosen for this study employ customer service and technical service representatives who provide 
service and support to either end (retail) customers or wholesale customers who both purchase and/or service the 
products or services provided by these organizations. In order to address the internal validity threat of location, all 
potential survey respondents were asked to complete the survey in the same type of environment (the workplace). In 
examining potential survey respondents, however, it was determined that not all had access to the Internet at work. 
For those who did, the Job Training and Job Satisfaction Survey was posted on an Internet survey website. Potential 
respondents with Internet access were e-mailed the survey link and asked to complete the survey online. Those 
without access were given paper copies of the survey to complete. 

A total of 301 customer/technical service employees completed the Job Training and Job Satisfaction survey, 
for an overall response rate of 55%. Broken down by methodology, Schmidt (2004) invited 263 employees to 
complete hard-copy surveys; 117 employees did so, for a response rate of 67%. Two-hundred-eighty -nine e-mail 
invitations were sent to potential online respondents, and 124 invitees completed the survey online, for a response 
rate of 43% (Schmidt, 2004). 

Workforce Development 

Strachota (2003) designed an original 4-point Likert survey instrument that was based on the typology of online 
interaction. Of interest in her research was identifying which interactions were most critical to online satisfaction. 
Constructs were developed based on the literature and identification of the four types of online interactions. 
Constructs for this instrument included: learner-content interaction, learner-instructor interaction, leamer-leamer 
interaction, learner-technology interaction, and general satisfaction. The construct of learner-technology interaction 
was measured using selected items from the Cassidy and Eachus (2000) survey. In order to establish content validity 
subject matter experts in the field of adult education and distance education as well as a panel of measurement 
experts examined these constructs, which included the definitions and questions for each construct. 

In order to ensure construct validity, a pilot study was conducted (N = 249). Through factor analysis the final 
instrument was reduced to twenty-seven items as six items had low factor loading that overlapped across all 
constructs, identifying that they were not good measures of that specific construct. The final instrument included 
seven items that were a measure of learner-content interaction, six items that were a measure of learner-instructor 
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interaction, eight items that were a measure of leamer-leamer interaction and six items that were a measure of 
general satisfaction. Removal of the six items resulted in moderate to high factor loading with the appropriate items 
loading within the specified construct. Therefore, questions within each construct were considered to have good 
internal validity. Reliability testing of the pilot study indicated a Cronbach’s alpha of .90 for the constructs of both 
learner-content interaction and general satisfaction. The constructs of learner-instructor interaction and leamer- 
leamer interaction resulted in a Cronbach’s alpha of .89. Items specific to learner-technology were not pilot tested 
as they had been piloted tested by Cassidy and Eachus (2000) and showed a Cronbach’s alpha of .97 for the single 
constmct of computer self-efficacy, which was equated as being synonymous with learner-technology for the 
purposes of this study. 

A convenience sample of 1,593 online students at a technical college in the Midwest area of the United States 
were given the Strachota Online Satisfaction Survey to assess the impact of different types of interaction relative to 
student satisfaction. Students were enrolled in various online courses as part of the program requirements for an 
associate degree in preparation for employment. The Strachota Online Satisfaction Survey was accessed through a 
web link in the college’s learning management system (LMS) that hosts all online courses. All survey participants 
had access to the Internet and all data were collected via an electronic format. A total of 849 students completed the 
Strachota Online Satisfaction survey for an overall response rate of 53.3%. 

Conclusion 

The two empirical cases illustrate the efficiencies that can be gained by using detailed processes of online survey 
development and distribution. Both the Schmidt (2004) and the Strachota (2003) online survey response rate 
exceeded the averages found by Sheehan (2001) who examined 3 1 online survey response rates from 1986-2000 and 
found the average response rate for those surveys to be 36.83%. Schmidt (2004) and Strachota (2003) employed a 
procedure for sending e-mail invitations to potential survey respondents. This procedure is regularly used by the 
Social and Economic Sciences Research Center, a research organization at Washington State University. The 
process consists of four contacts via e-mail. Sent first was a pre -message, requesting potential respondents to watch 
for the survey link, followed by a message that contained the link, along with survey instructions, followed by an e- 
mail thank-you note or a reminder, which also included the link, and a final reminder, also with a link to the survey 
(Schaefer & Dillman, 1998). Schmidt (2004) conducted this procedure over a two-week time period whereas 
Strachota (2003) conducted this procedure over a 4-week period of time. 

Although conducting online surveys is far more efficient and cost effective than mail or telephone survey 
distribution, knowledge and application of online survey software as well as demonstrating the necessary technology 
skills is required for this methodology. The efficiency and cost effectiveness of conducting online survey research, 
however, far outweigh the time and efforts required to research and learn a specific software program application. 

Distributing a survey through a link via e-mail or a Learning Management System is far more time efficient 
than printing and mailing individual surveys or using the telephone for conducting 1 : 1 survey responses. Regardless 
of the survey method used, however, all data that are collected must be entered into a statistical program for 
analysis. Survey data gathered online offers the researcher an advantage in that it can be automatically downloaded 
to a statistical program for analysis. This method allows for tremendous time saving when compared to manual data 
entry, which must be done for hard-copy surveys. Furthermore, the possibility for data entry errors is minimized 
when compared to the manual entry of data. These are two examples that illustrate the high level of efficiency (or 
high ratio of output to input) associated with online survey distribution and data management. 

The cost of survey software is also far less costly than traditional methods of using telephone or mail 
distribution (Strachota et ak, 2005). Using online surveys is a prime example of what Pande, Neuman, and 
Cavanagh (2000) referred to as an efficient process, or one that uses “less money, time, and materials” (p. 203). 

Recommendations for the Use of Online Snrveys 

The process of using online surveys for collecting data to measure satisfaction in job training and workforce 
development in the two empirical studies described above allowed the researchers to identify some advantages, 
limitations, and suggestions for the future use of online surveys. It is important to consider the pros and cons 
associated with the use of online surveys, as well as the tips for survey development and distribution, before 
undertaking a project of this nature. 

Advantages of Online Surveys 

There are advantages to using online surveys to gather data. When compared with traditional mail or telephone 
surveys, Schonlau, Fricker, and Eliott (2001) note that “sampling error issues with Internet surveys are generally the 
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same as those with eonventional surveys” (p. 29). Sehonlau et al. (2001) state that Internet surveys may be 
preferable to mail or telephone surveys when the survey eontains questions of a partieularly sensitive nature. The 
use of Internet-based surveys may allow the program developer to delve deeper into evaluation-related topies, and 
may result in more honest feedbaek than would traditional survey methodologies. 

Advantages of online survey distribution inelude inereased time effieieney, deereased data entry error, inereased 
item response rate, and deereased eost. Due to available teehnology, administration of online surveys has resulted in 
inereased time effieieney in both distribution and reeeiving of survey results. The majority of eolleges, universities, 
and businesses today that deliver distanee learning eourses and programs have invested in a LMS. The LMS is 
designed to allow for the posting of an announeement with a direet web link to an online survey instrument. The 
LMS offers effieieney for students who are enrolled in a eourse beeause they ean aeeess the survey instrument and 
eomplete it at a eonvenient time. The LMS also eontains a database of e-mail addresses for all students. When the 
LMS is not available the proeess beeomes far more time-eonsuming as e-mail addresses must be obtained before a 
mass mailing ean be eondueted. Both of these methods, however, still remain far more time effieient than telephone 
or mail surveys. 

Deereased data entry error is an important faetor when eondueting survey researeh. Traditional telephone and 
mail surveys require manual data entry. Eleetronie survey distribution does not require manual entry as data are 
eolleeted through a given software program and then downloaded into a statistieal program sueh as SPSS for 
detailed analysis. This feature saves time, is more aeeurate, and is espeeially useful when eolleeting data with a large 
sample size. 

Inereased item response rate ean also be eontrolled through eleetronie surveys, as most of the software 
programs ean be set so that all items must be eompleted before submission. Traditional mail surveys often are 
returned with data being absent in given fields. Further, qualitative item response rates ean be inereased due to 
legibility of eleetronie responses versus oeeasional illegible mail responses. Sehaefer and Dillman (1998) found that 
eleetronie surveys resulted in a quieker return rate, a slightly lower item non-response rate, and more eomplete 
answers to open-ended questions. 

Another advantage of eleetronie survey distribution is deereased eost eompared to telephone and mail 
distribution. A major eoneem of survey researeh is the potential for high non-response error. In order to minimize 
non-response error, multiple eontaets are made to have partieipants eomplete a given survey. Sehaefer and Dillman 
(1998) support the use of multiple eontaets as a method for survey dissemination as it inereases the response rate. 
Informing partieipants that a survey will be fortheoming through an announeement, presenting the aetual survey 
link, a thank-you or reminder e-mail with the survey link, as well as a final e-mail reminder with the survey link ean 
all be entered in an LMS and/or e-mailed at no eost. Contaeting partieipants multiple times through mail or 
telephone distribution, however, is far more eostly and time eonsuming. Eleetronie survey distribution, therefore, is 
a far more effeetive method for data eolleetion and evaluation of edueational and training programs. 

Limitations of Online Surveys 

The online survey must be designed so that it has user-friendly navigation. Certain eomputer skills sueh as seleeting 
radio buttons, use of drop-down menus, and serolling are eommonly required skills. Correetly reading the direetions 
and paying attention to detail is also important. Dillman and Bowker (2001) identify the need to provide speeifie 
direetions for eaeh required eomputer aetion as well as any other neeessary instruetion to avoid frustration and 
termination of the survey before eompletion. 

A potential limitation of e-mail survey researeh ineludes the diffieulty in sorting out responders versus non- 
responders, espeeially with a large sample size. Oeeasional annoyanee may be pereeived by reeeiving multiple e- 
mails requesting a response. E-mail, like mail delivery, may be pereeived as junk mail and deleted without 
responding. In order to aehieve higher eomplianee among potential responders, it is suggested that some ineentive 
be established as part of the proeess. 

It is important to reeognize that eondueting online surveys versus telephone or mail surveys is a good fit for 
those who use a eomputer as a eommon mode of eommunieation as well as for those who enroll in online learning 
for eareer and job advaneement. While eondueting online surveys is far more effieient and eost effeetive it is 
important to reeognize the audienee being surveyed. Although we live in a teehnologieal age, we are still faeed with 
the issue of eomputer aeeess. While aeeess has inereased sueh that 204.3 million Amerieans have Internet aeeess in 
their home, 25% of Amerieans still do not have Internet aeeess (Kim, 2004). Timmerman (2002) suggests that 
“before assuming web-based surveys are the most advantageous method for surveying a target population, it should 
be determined if those advantages exist within that population (p. 5). 

Online Survey Distribution Recommendations 

One area that is eritieal in the proeess of data eolleetion is survey distribution. Online survey distribution typieally 
involves the use of an e-mail message sent to potential respondents, whieh ineludes a link to the survey’s Uniform 
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Resource Locator (URL). Following are some recommendations for effective and efficient survey distribution using 
e-mail: 

• E-mail address and list management is important. Be sure everything in all potential respondents’ e-mail 
addresses is spelled correctly. This will reduce the number of undeliverable messages to be corrected and 
resent. 

• Develop a method to track whether or not all potential respondents received the e-mailed survey invitation. 
Send test messages to a variety of people with different e-mail addresses (i.e., business or organizational 
addresses and personal addresses) before sending to potential respondents. 

• Creative and informative information in the e-mail subject line will prevent potential respondents from 
thinking that your e-mail message is spam or junk mail, and simply delete it. 

• E-mail filters used by many businesses and individuals may prevent e-mail messages from getting to 
potential respondents. Online survey websites such as those mentioned earlier have the advantage of 
allowing the researcher to send e-mail survey invitations to many people at once. However, e-mail filters 
may recognize the return address (which is often the survey website) as spam. This may cause the survey 
invitation to be diverted from potential respondents. 

• Ideally, survey respondents will access a web-based survey by clicking on a URL link in the body of the e- 
mail. Be sure all links work before sending them to potential respondents. Testing e-mail messages can be 
effective in insuring a link works. 

• Have a backup to the URL link. Instruct survey respondents that if the link does not work, they can copy 
the link and paste it into the web address line. Also, provide instructions for accessing the survey by 
providing the website address to the online survey tool, along with instructions for respondents who go 
directly to the website. 

• E-mail messages and survey links should be thoroughly tested before any survey invitations are sent to 
potential respondents. Send test messages to as many different types of e-mail addresses as possible. Along 
with those test messages, send links to test surveys on the survey website. This, again, will insure that as 
many respondents as possible receive the message, access the online survey using the link provided, and 
complete the survey (Schmidt, Strachota, & Concei9ao, 2005). 
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